Natural-Gradient Actor-Critic Algorithms

We prove the convergence of four new reinforcement learning algorithms based on the actorcritic architecture, on function approximation, and on natural gradients. Reinforcement learning is a class of methods for solving Markov decision processes from sample trajectories under lack of model information. Actor-critic reinforcement learning methods are online approximations to policy iteration in which the value-function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient ascent/descent. Reinforcement learning methods based on policy gradients in this way are of special interest because of their compatibility with function approximation methods, which are needed to handle infinite or large state spaces, and the use of temporal-difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the estimates. Natural or Fisher-information versions of policy gradient algorithms are of interest because they can produce better conditioned parameterizations and have been shown to have further reduced variance in some cases. Our results extend prior two-timescale convergence results for actor-critic methods by Konda and Tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients. Our results extend prior empirical studies of natural-gradient actor-critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms. We present empirical results verifying the convergence of our algorithms. A limitation of our results is that they do not address the use of least-squares methods and eligibility traces.